The present invention relates to virtual studio systems, particularly but not exclusively chroma-keying studio systems.
Chroma-keying is a technique that allows synthetic images to be created by determining if a pixel in a camera image belongs to foreground or background based on colour. The foreground pixels can then be keyed into a second image, e.g. a computer-generated or static background as in virtual studios. The background in the studio must have the particular keying colour, typically blue. This is usually achieved by painting the studio floor and visible walls with the keying colour. Another option is to use a retro-reflective cloth and an active illumination ring around the camera lens as proposed in our earlier GB-A-2,321,814 the entire disclosure of which is incorporated herein by reference.
Although Chroma-keying is a preferred application, the present invention is also applicable to other keying techniques (for example using temporally modulated lighting of the background to distinguish foreground and background).
In this specification, the term “participant” is intended to encompass without limitation any actor or presenter (human or animal) or indeed any form of animate entity which might benefit from a view of a virtual scene.
One problem for a participant in a virtual studio is to interact with the objects in the virtual set that are not visible for him or her. In particular, if the participant has to interact with a virtual object, e.g. a virtual character, then there is often a difference between the direction in which the participant looks, and the position of the virtual character that he should be looking at (the so-called eye-line problem). This can be quite disturbing in the final programme.
An approach to providing the participant with a visual cue of objects in the virtual set is described in U.S. Pat. No. 5,886,747. This discloses projecting an outline of basic objects onto the floor and walls and further describes how to make these marks invisible to a studio camera by projecting only during the blanking interval of the camera. However, the method described is restricted to showing the main points of intersection of the important virtual objects with the particular floor or wall so, for example, a virtual actor in the scene would only be visualised as footprints. It is disclosed that the system can typically draw 100 dots per frame which is considered sufficient to outline large static objects or to animate small objects. Indeed, the solution can effectively be considered an electronic alternative to the previous measure of physically placing masking tape on the floor and walls at important object boundaries (as discussed in column 1, lines 53 to 57 of the document).
This prior art visual cue system for virtual studios does not however deal directly with the eye-line problem, i.e. the case that a participant has to look towards a portion of a virtual object that is in space (for example the face of a virtual actor) but merely gives the participant an idea of the position of objects in the studio.
By way of general background, outside the field of virtual studio production, systems are known that can provide functionality to enable a user to see objects in a virtual environment, but these are generally neither intended nor suited to the constraints of a virtual studio environment. One example is projection-based VR (virtual reality) systems like the CAVE: Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V., and Hart, J. C. “The CAVE: Audio Visual Experience Automatic Virtual Environment,” Communications of the ACM, Vol. 35, No. 6, June 1992, pp. 65-72. This system tracks the position of the viewer's head and computes an image for that particular view-point of the viewer, which is then projected onto a large screen forming one wall of the environment. Several screens are usually used to provide a wide field-of-view, and some installations provide an all-round view using six projection screens that completely surround the viewer. Therefore it is possible to present objects that appear virtual in space. The head position of the viewer is usually tracked using a special device, e.g. a helmet with an electromagnetic transmitter. Such a system is of course not designed for nor suited to a virtual studio environment for a number of reasons. The head-tracking device would be visible in any final programme and, more fundamentally, virtual studios which may have a combination of “real” objects at arbitrary positions and require a background of a key colour do not provide the open space with large screens on which images can be projected at will as this system requires.